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FINAL  REPORT  ON  A_ SERIES  OF  STUDIES  ON  THE  DESIGN  REQUIREMENTS 
FOR  A  REAL-TIME  INFORMATION  AND  COMPUTATIONAL  SYSTEM  TO  ASSIST 
R&D  MANAGEMENT  IN  THEIR  PROJECT  SELECTION  DECISION-MAKING 

Purpose 

The  purpose  of  this  paper  is  to  report  on  a  series  of  studies  currently  underway 
at  Northwestern  University.  The  studies  are  part  of  a  long  term  project  aimed 
at  specifying  the  design  requirements  for  a  real-time, 'computer-aided,  information 
system  for  the  selection,  review,  and  evaluation  of  research  and  development 
(R&D)  projects.  Specifically  the  report  focuses  on  phases  of  the  research  which 
have  been  supported  by  the  Office  cf  Naval  Research.* 

Scope  of  the  Study 

Since  1959,  Northwestern  University  has  had,  under  the  direction  of  Professor 
Albert  H.  Rubenstein,  a  Program  of  Research  on  the  Management  of  Research  and 
Development.  The  program  consists  of  approximately  a  dozen  individual  projects 
or  subprograms,  focused  on  specific  aspects  of  the  R&D  process.  Among  the  sub¬ 
programs  (each  consisting  of  a  continuing  series  of  staff  sr-.1i.es  and  graduate 
theses)  are  studies  of  the  effects  of  decentralization  on  R&D,  organization  of 
R&D  in  newly  developing  countries,  the  generation  n  ■  flow  of  ideas  in  R&D,  the 
information- seeking  behavior  of  researchers,  and  project  selection  in  R&D. 

The  latter  study  is  the  subject  of  this  paper.  This  study,  which  focuses  on  the 
project  selection  process  in  R&D,  was  formally  initiated  in  1965.  It  has  four 
interrelated  objectives:  ** 

1.  Improve  the  present  understanding  of  the  informacion  requirements  for 
improved  project  selection  and  review  in  R&D. 

2.  Analyze  top  management's  information  requirements  and  criteria  for 
project  selection  and  approval. 

3.  Test  and  evaluate  formal  information  systems  and  models  (generally 
mathematical)  that  have  been  proposed  in  the  literature  for  project  selection. 

4.  Attempt  to  specify  the  requirements  for  a  useful  real-time  information 
system  for  project  selection  and  review. 

In  pursuing  these  objectives,  studies  have  been  undertaken  at  three  large  indus¬ 
trial  organizations  and  at  two  large  governmental  R&D  organizations. 

*  Office  of  Naval  Research,  Research  Grant  Nom--l 228-638  from  1965-1969 
**  Source:  Original  proposal  to  ONR  (Our  document  number  65/24) 


A  Premature  Termination  at  One  Site 

A  substantial  amount  of  our  research  effort  in  one  large  industrial  R&D  facility 
was  limited  to  the  development  of  a  methodology  for  studying  the  project  selec¬ 
tion  process.  Some  of  the  methodological  problems  involved: 

1.  Identifying  events  which  are  often  diffused  throughout  on  organization 
and  which  are  spread  over  an  extended  time  period. 

2.  Identifying  individuals  who  are  involved  in  these  various  "project  selection"  event 

3.  Obtaining  actual  criteria  *Aiich  decision  imlcers  use  during  these  events. 

4.  Reconstructing  the  information  state  of  decision  makers  at  the  time  of 
these  events. 

Some  of  the  procedures  and  techniques  developed  during  this  study  reflect  a 
conscious  attempt  to  minimize: 

1,  The  distortion  in  the  study  data  due  to  a  lag  between  the  occurrance 
of  an  event  and  the  recording  of  data  relevant  to  the  event. 

2.  The  annoyance  of  the  people  being  sampled. 

3.  The  effect  on  behavior  resulting  from  asking  subjects  to  examine  fre¬ 
quently  their  actual  behavior. 

4,  The  expenditures  necessary  to  achieve  an  acceptable  number  of  field 
observations. 

* 

Unfortunately,  much  of  this  investment  in  methodology  could  not  be  recouped  at 
this  particular  field  site  as  the  laboratory  management,  found  that  they  were  unwilling 
to  continue  their  participation  into  the  experimental  phase  of  the  study.  Accordingly, 
further  research  at  the  site  was  prematurely  terminated  (68/D,  67/2)*. 
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A  Series  of  Studies  at  One  Site 

At  another  large  manufacturing  firm,  three  studies  were  undertaken.  These 
studies  focused  upon  the  project  selection  decision  and  the  procedures  by  which 
projects  are  evaluated  (69/23,  68/2,  67/55,  67/45). 

In  the  first  study  the  stability  and  sensitivity  of  three  economic  decision  models 
was  investigated.  Tests  were  performed  on  a  portfolio  of  51  projects  undertaken 
by  the  firm.  Three  vectors  were  considered:  research  expenditures,  capital 
investments,  and  excess  value  anticipated  for  each  of  the  51  projects.  Each 

r-lhese  numbers  refer  to  our  program  documents.  A  listing  of  these 
documents  is  attached. 
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vector  was  examined  at  each  interval  of  a  10  year  planning  horizon.* 

The  decision  models  investigated  were  (1)  a  marginal  efficiency  of  capital  model, 

(2)  an  excess  current  value  model,  and  (3)  an  excess  terminal  value  model.  The 
results  indicate  that  the  models  were  stable  and  sensitive  to  systematic  varia¬ 
tions  of  (a)  anticipated  excess  value  of  each  project  over  their  planning  hori¬ 
zons,  (b)  the  length  of  the  planning  horizon,  (c)  the  reinvestment  rates  of 
interest  associated  with  each’ planning  interval,  (d)  the  capitalization  rates  of 
interest  (i.e.,  the  opportunity  cost  for  investment  resources)  relevant  to  the 
planning  intervals,  and  (e)  the  method  used  to  adjust  for  risk. 

In  a  second  study  at  this  field  site  an  attempt  was  made  to  study  the  relation 
between  analysis  and  decision,  or  between  analysts  and  decision  makers.  The 
second  study  differs  from  the  first  study.  The  first  study  sought  to  examine, 
in  an  empirical  context,  the  logio  implicit  in  three  economic  decision  models. 

The  second  study  sought  to  situate  this  logic  in  relation  to  the  actual  decisions 
made  by  the  organization's  decision  makers.  Specifically,  reasons  for  the  divergence 
between  the  results  of  the  analysis  and  the  results  of  the  actual  decision  making 
process  were  hypothesized  and  tested. 


The  predictive  criteria  for  this  divergence  were  (1)  the  perceived  long  term 
contribution  to  the  economic  value  of  the  firm,  (2)  the  perception  of  a  relative 
priority  of  project  importance,  and  (3)  the  perceived  project  urgency.  Additional 
perceptual  data  was  gathered  on  criteria  which  were  not  found  to  contribute  to 
the  explanation  of  this  divergence.  These  criteria  included  (1)  the  amount  of 
investment  resources  previously  consumed  by  the  project,  (2)  whether  the  project 
was  in  the  exploratory,  developmental,  or  design  phase  of  research,  (3)  the 
laboratory  in  which  the  research  activity  was  being  performed,  and  (4)  the  ap¬ 
parent  interest  of  key  members  of  top  management. 


*  For  this  study  each  project  was  provisionally  assumed  to  be  independent  and 
to  compete  individually  for  funding  or  continued  funding  from  an  unlimited  fund 
of  research  investment  resources.  The  consequences  of  relaxing  these  assumptions 
are  still  being  investigated. 
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A  third  and  final  study  at  this  field  site  attempted  to  focus  upon  scoring 
models  and  preference  functions  ns  a  means  to  discriminate  between  projects. 

For  this  study  a  scoring  model  was  used  at  the  laboratory  level  and  the  prc- 
dective  ability  of  the  model  was  assessed  in  light  of  a  final  project  selection 
decision  made  at  the  firm's  corporate  headquarters. 

The  overall  rationale  for  assessing  the  predictive  ability  of  scoring  models  is 
that  inherent  in  the  present  structure  of  such  models,  no  attempt  is  made,  in  gen¬ 
eral,  to  specify  a  system  which  describes  the  behavior  of  a  decisionmaker^  even 
though  portions  of  such  a  system  are  implied  by  the  way  project  scores  are 
achieved.  Specifically,  it  can  be  argued  that  scoring  models  state  the  form  of 
a  preference  function  as  an  assumption  rather  than  deriving  it  from  a  set  of 
behavioral  axioms. 

The  model  used  in  this  study  required  subjective  estimates  by  one  or  more 
evaluators  of  six  different  project  attributes.  A  project  score  which  was  to  be 
the  product  of  the  average  attribute  scores  for  each  project  was  calculated. 

These  project  scores  were  then  to  be  used  to  evaluate  and  rank  the  projects. 

The  results  of  this  study  indicate  that  the  project  scores  (i.e.,  the  product  of 
the  average  attribute  scores)  do  not  satisfactorily  predict  the  final  project 
selection  decision. 

Alter  ate  scoring  systems,  based  on  discriminant  analysis,  were  developed  in  an 
attempt  to  identify  which  combination  of  project  attribute  scores  would  yield  a 
project  score  such  that  a  satisfactory  prediction  of  the  final  project  selection 
decision  could  be  made.  Two  scoring  systems  were  found  to  significantly  im¬ 
prove  predictability.  In  addition,  both  of  these  new  scoring  systems  resulted 
in  a  notable  reduction  in  the  required  data. 

An  Experiment  with  a  Pro  icct  Selection  Process 


At  a  large  capital  goods  manufacturing  firm  an  experimental  change  to  one  of 
their  R&D  project  selection  processes  was  undertaken  and  evaluated. 
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Work  at  this  field  site  began  wtth  frequent  open-ended  interviews  between  the 
company's  director  of  exploratory  research  and  our  investigator.  Twenty  to 
thirty  minute  telephone  interviews  were  held  regularly  once  or  twice  a  week  for 
approximately  three  months.  These  were  complemented  by  monthly  face-to-face 
interviews.  The  purpose  of  these  discussions  was  to  identify  the  criteria 
employed  by  this  decision-maker  in  arriving  at  project  selection  decisions.  In 
a  project-by-project  review  of  the  1967  portfolio,  variables  consistently  con¬ 
sidered  in  the  decision  maker's  evaluation  thought  processes  were  elicted.  Sub¬ 
sequently,  a  form  was  developed  which  incorporated  these  criteria  into  a  formal 
project  selection  model. 

As  the  first  phase  of  the  study  progressed,  a  concurrent  effort  was  begun  to 
develop  a  computer-based  project  selection  technique.  The  program  essentially 
combined  probability  estimates  associated  with  possible  outcome  values  assigned 
to  each  of  the  criteria  for  projects  in  the  decision-maker's  portfolio.  Using  a 
Monte  Carlo  simulation  technique,  the  input  values  and  probabilities  specified 
by  the  decision-maker’s  staff  provided  an  output  which  allowed  them  to  identify 
explicitly  the  risk  associated  with  potential  project  pay-offs.  Together,  the 
input  format  and  the  computer  program  were  designed  to  help  the  decision-maker; 

1.  consider  a  wide  range  of  possible  outcomes  for  a  given  project, 

2.  interrelate  various  items  of  information  relevant  to  his  analysis  of  a 

.  particular  program,  and 

3.  appreciate  the  uncertainty  inherent  in  each  project,  as  well  as  facil¬ 
itate  the  pinpointing  of  those  elements  most  likely  to  "make  or  break" 
a  project. 

As  part  of  a  third  phase,  the  decision-maker  was  able  to  use  the  input  format 
and  computer  program  to  evaluate  31  projects  accepted  for  funding  in  1969.* 
Although  this  stage  was  aimed  primarily  at  pilot  testing  and  installation  of  a 
project  selection  system  for  later  experimentation,  a  considerable  amount  of 
worthwhile  data  were  obtained  concerning  the  consequences  of  introducing  a  formal 
project  selection  technique. 


*  The  decision-maker  held  small  project  review  meetings  in  which  a-8  persons 
responsible  for  a  project  ware  asked  to  jointly  supply  necessary  data. 
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For  example,  nearly  all  of  the  participants  from  the  decision-maker's  staff 
welcomed  the  opportunity  to  explicitly  indicate  the  uncertainty  in  their  esti¬ 
mates  through  the  mechanism  of  associating  probabilities  with  projected  outcomes. 

A  significant  percentage,  however,  encountered  extreme  difficulty  in  translating 
their  intuitive  feelings  into  actual  estimates. 

Also,  there  were  some  so-called  "eye-openers"  concerning  the  consequences  of 
several  projects.  Two  projects  were  dropped  on  the  recommendations  of  the 
individual  proposers  in  light  of  their  analysis  using  the  new  project  selection 
system.  Another  project  research  strategy  underwent  serious  revision  based- 
upon  the  results  of  several  sensitivity  analyses  made  possible  by  the  computer¬ 
ized  technique. 

Next,  the  new  procedure  was  experimentally  incorporated  into  the  company's 
actual  exploratory  research  project  selection  decision  process.  In  total,  69 
persons  participated  in  a  total  of  105  evaluations  using  the  new  procedure. 

Each  evaluation  required  1  1/2  to  2  hours  from  each  participant. 

Finally,  64  of  the  69  persons  using  the  new  procedure  were  interviewed  and  asked 
to  complete  a  questionnaire.  This  data  was  used  to  assess  their  willingness  to 
adopt  the  new  procedure  and  served  as  a  basis  for  pursuing  reasons  for  this 
willingness  or  lack  of  willingness  to  adopt  such  a  procedure. 

The  results  of  this  study  indicate  that  there  were  at  least  three  major  determinants 

of  an  individual's  attitude  towards  the  use  of  the  procedures.  These  variables 
involved  the  individual's  perceptions  of: 

1.  the  value  of  the  data  generated  by  the  technique, 

2.  the  appropriateness  of  the  information  considered  by  the  organization 
during  the  project  review  process,  and 

3.  the  value  of  changes  in  projects'  research  strategies,  attributable  to 
the  use  of  the  formal  technique. 

The  results  further  indicate  that  an  individual's  perceived  need  for  additional 

information  was  greatly  increased  because  the  technique  was  used  and  that  individ¬ 

uals  indicated  that  they  acquired  more  information  because  the  technique  was 
used.  However,  27  of  the  64  respondents  indicated  that  their  acquisition  of  ad¬ 
ditional  information  was  seriously  inhibited  by  the  lack  of  an  appropriate 
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mechanism  for  acquiring  information.  Many  of  these  respondents  attempted  to 
acquire  additional  information  and  found  that  relevant  information  did  not  exist. 
Others  found  difficulty  obtaining  information  due  to  the  difficulties  in  cross¬ 
ing  organizational  boundaries  and  a  general  unwillingness  to  part  with  poten¬ 
tially  relevant  data  by  those  who  were  presumed  to  have  the  necessary  data. 

Some  Specif  teat  Ions  for  a.  Real-Time  Information  System 

Approximately  40  managers  of  R&D  organizations  have  cooperated  with  us  in  an 
attempt  to  set  forth  some  preliminary  specifications  for  a  real-time  system. 

These  managers,  working  in  one  of  four,  three  month  seminars  on  R&D  management, 
have  prepared  design  specifications  for  an  information  and  computations  jl  system 
that  would  be  of  value  to  their  organizations. 

Drawing  upon  this  resevoir  of  experience  and  data,  two  studies  were  initiated. 

The  first  study  attempted  to  identify  the  basic  structure  for  a  system  which 
would  satisfy  all  or  part  of  the  requirements  set  forth  by  this  group  of  managers. 
In  addition,  an  attempt  was  made  to  compare  the  objectives  and  requirements  among 
the  individual  systems. 

Results  from  the  first  study  indicate  that  a  batch  processing  system  would 
satisfactorily  accommodate  the  immediate  needs  of  these  managers.  Regarding  the 
internally  generated  data  (e.g.,  project  costs,  progress,  and  budgeting  infor¬ 
mation)  the  basic  data  structure  and  process  requirements  were  in  general 
uniformly  and  clearly  defined.  Similarly,  the  requirements  for  external  data 
(e.g.,  market  data)  were  in  general  uniformly  and  clearly  defined.  However,  the 
requirements  necessary  to  effectively  relate  these  two  sets  of  data,  and  items 
within  these  packages,  could  not  be  adequately  specified. 

The  second  study  attempted  to  examine  some  of  the  problems  associated  with  in¬ 
corporating  soft,  or  somewhat  less  factual  data,  into  an  information  system. 

This  data  included  such  items  as  management's  Insights  into  causal  relationships 
in  the  market  place  and  management's  judgments  concerning  the  probability  of 
uncertain  events. 

The  results  of  this  study  indicate  that  such  organizational  functions  as  assess¬ 
ing  the  relevancy  of  data  for  a  particular  decision  situation  and  the  function 
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of  deliberating  over  a  best  course  of  action  among  alternatives  must  be  given 
further  consideration.  The  first  function  was  further  identified  as  one  which 
would  involve  the  allocation  of  responsibility  for  validating  and  packaging 
relevant  information.  The  second  function  was  further  identified  as  one  which 
would  involve  processes  that  might  result  in  a  more  systematic  ordering  of 
management’s  deliberation  process. 

Progress  at  Two  Government  Facilities 

Two  studies  are  in  progress  at  two  government  R&D  facilities.  The  study  at  the 
first  site  has  been  limited  to  descriptive  aspects  of  the  project  selection 
process  while  the  study  at  the  second  site  has  been  directed  toward  the  develop¬ 
ment  of  a  series  of  experiments  with  the  project  selection  process. 

As  part  of  the  first  study,  a  set  of  data  has  been  acquired  which  comprises 
laboratory  reports  and  review  sheets  for  all  projects  proposed  in  that  facility, 
over  a  period  of  four  years.  This  documentation  represents  the  core  information 
to  which  certain  criteria  are  applied  in  arriving  at  project  selection  decisions. 
In  addition,  a  set  of  instruments  has  been  designed  for  further  research  in  this 
laboratory.  This  instrument  package  has  been  designed  for  the  purpose  of  mon¬ 
itoring  such  aspects  of  the  project  selection  process  as  the  time  at  which 
decisions  are  made,  the  content  of  the  decisions,  and  the  criteria  and  premises 
upon  which  decisions  are  based. 

The  second  study  was  undertaken  with  the  intention  of  designing  a  series  of  field 
experiments.  This  series  of  experiments  will  involve  assessing  the  relative 
adequacy  of  a  number  of  formal  project  selection  and  allocation  methods.  This 
assessment  will  be  made  by  including,  on  an  experimental  basis,  relevant  methods 
into  the  organization's  R&D  project  selection  process. 

At  present,  in  depth  interviews  with  responsible  decision  makers  have  been  com¬ 
pleted  and  an  attempt  is  being  made  to  make  a  preliminary  statement  of  the 
decision  problem.  Concurrently,  the  feasibility  of  evaluating  several  model 
forms  are  being  pursued.  The  model  forms  include  constrained  capital  budgeting 
models,  modified  scoring  models,  and  heuristic  decision  rules. 
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Some  Tcntat  jyx'  Conclusions 

The  results  of  this  research  project  indicate  that  much  of  the  information  re¬ 
quired  for  either  a  descriptive  or  prescriptive  analysis  of  the  project  selection 
process  is  not  collected  or  even  recorded  by  most  R&D  organizations .  Consequent¬ 
ly  a  variety  of  methods--interviews  ,  observations,  and  questionnaires--must  be 
used  to  obtain  appropriate  information.  Accordingly,  the  development  of  means  of 
collecting,  reducing,  and  evaluating  these  kinds  of  data  must  continue  to  be  an 
integral  part  of  our  research  activities. 

Our  research  on  scoring  models  and  preference  functions  indicate  that  such  model 
forms  do  not  offer  a  satisfactory  means  for  discriminating  among  research  pro¬ 
jects.  This  research  has  led  to  a  realization  that  attempts  must  be  made  to 
further  examine  the  methodological  and  theoretical  structures  upon  which  present 
models  arc  based. 

Results  further  suggest  that  prescriptive  and  descriptive  models  of  the  R&D 
project  selection  process  must  take  explicit  account  of  the  probabilistic  nature 
of  the  R&D  process.  Accordingly,  our  future  efforts  will  be  concerned  with  the 
treatment  to  be  given  probabilistic  data  that  can  be  collected  and  its  relation¬ 
ship  to  the  data  called  for  by  formal  decision  models. 

Our  experiments  with  the  use  of  formal  project  selection  techniques  in  organi¬ 
zations  has  led  us  to  believe  that  the  use  of  such  techniques  may  result  in 
changes  to  a  number  of  organization  and  organizational  variables.  For  instance, 
our  results  suggest  that  new  and  different  information  is  considered  by  an 
organization  when  a  formal  decision  model  is  used  and  that  the  objectives  and 
activities  associated  with  a  research  project  can  also  be  affected.  Our  plan  is 
to  further  examine  the  consequences  of  using  formal  decision  models  upon  such 
variables. 

Finally,  our  exploration  of  the  design  requirements  for  a  real-time  information 
and  computational  system  to  assist  R&D  management  in  their  project  selection 
decision-making  will  continue  to  be  pursued  along  the  dimensions  already  dis¬ 
cussed.  However,  our  research  further  indicates  a  need  to  investigate  the 
problems  related  to  the  estimating  of  project  attributes.  Accordingly,  a  scries 
of  studies  are  being  planned  with  the  objective  of  examining  problems  related 
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to  estimation  (69/D).  Some  of  the  questions  asked  in  this  study  are:  What  is  the 
relative  importance  of  those  factors  judged  to  be  relevant  to  the  technical  suc¬ 
cess  of  a  research  project?  What  factors  appear  to  best  characterize  estimates 
made  by  individual  estimators?  What  factors  appear  to  characterize  the  relia¬ 
bility  of  an  estimator? 
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